Conversations with Non-Human Actors in E-Democracy Evaluation

Software systems that structure conversation (and vice versa), formalise the roles of interactors, process their interactions, and provide other forms of ‘support’ are the object and subject of design and evaluation in CSCW, and its related fields of research and practice. E-democracy is a relatively recent addition to them, and can be characterised as the use of network technologies to promote collaboration between actors for policy-making purposes, whether acting as citizens, their elected representatives, or on behalf of administrations, parliaments or opposition groups. This paper explores the applicability to edemocracy evaluation of sociological approaches to conversation that have been widely used in CSCW studies to demonstrate how relations between human and non-human actors are constructed in online and offline settings. Two brief examples are given from the evaluation of software in the projects EDEN and AVANTI, under the European Commission’s programme of research and development of Information Society Technologies. In both projects ethnographic analysis has had a role in evaluating the risks and benefits of software that mediates conversation between citizens and administrations. These examples are used to argue for the value of ‘pragmatist/culturalist’ approaches in e-democracy evaluations, which may otherwise lack the richness of CSCW studies of technology usage. Such approaches demonstrate how edemocracy development entails ‘politics’ in the re-allocation of representational roles from human actors to software ones. Attending to the detail of conversation between actors, human or otherwise, can help to make design and deployment choices more transparent than they would otherwise be. Introduction: Framing E-democracy as Conversation CSCW research has for several decades dwelled in the ‘great divide’ between social and technical perspectives, which the more recent development of e-democracy research sits astride. Many governments are actively experimenting with web applications that are intended to enhance public participation in policy-making. These are often ‘vertical’ applications of generic tools. For example discussion fora are being widely deployed by local, national and international institutions to consult their constituencies on issues and policy drafts, and by non-governmental organisations to articulate positions and influence agendas (see e.g. OECD, 2003). Yet the impact of these fora on policy-making is far from obvious. The claimed benefits of enhanced deliberation, more transparent policy-making, and wider participation by citizens are difficult to ascertain since e-democracy experiments are frequently marginal and ineffectively linked to existing decision-making practice (Coleman and Goetze, 2001). The ‘fit’ between e-democracy systems and policy-making practice is thus a critical issue for ensuring the adaptability of e-democracy systems (Westholm, 2003), an issue long addressed in CSCW studies that investigate the fit between working practices and collaborative technologies. E-democracy framed as ‘conversation’ thus has a wider scope than the usability or acceptability of discussion fora, embracing other online and offline communications between citizens, their elected representatives, and public administration officers. Many writers on politics and technology are interested in the mediation of that communication by civic (and othe r) organisations, and by concepts of citizenship, and in characterising their relationships vis-à-vis the state as alliances, partnerships, lobbies, oppositions and so forth. E-democracy research thus cuts across the entire range of concepts and methods used to such ends in organisation science (see e.g. Morgan,). Relatively fewer writers have attended to the messy detail of conversations in and around e-democracy systems, although such studies continue to demonstrate their value in CSCW (Martin et al, 2001). CSCW@E-democracy Workshop, ECSCW ’03, Helsinki Sept. 15 2003 2 The lack of such studies in e-democracy is despite the parallels between the development of e-democracy systems and inter/intra-organisational internet/intranet developments. CSCW has tended to focus on the latter, and on the application of social science methods to studies of working practice and system usage that are meant to inform systems design, often for the purposes of requirements analysis or evaluation (e.g. Hughes et al, Crabtree, 2000). The methodological developments have tended to be cast in terms of a distinction between (on the one hand) managerial notions and models of organisational structures, cognitive and business processes, mainstream sociological concerns with developing or applying theoretical concepts of social structure, and (on the other) ethnographic approaches informed by ethnomethodology (e.g. Suchman 1987), interactionist sociology (e.g. Wenger, 1997) and actor-network theory (e.g. Suchman, 2001). It is mainly the latter, ‘pragmatic/culturalist’ approaches (Kaghan and Bowker, 2001) that this paper is concerned with, since they attend to the detail of conversation as working practice, in order to subject design concepts and prototypes to a ‘reality check’ that can demonstrate their fit with practice, or otherwise. The paper is based on the premise that this is necessary in e-democracy systems, and moreover that CSCW offers the latter perspectives and technologies of accountability (Suchman, 1994), that can both enrich e-democracy systems development and address its frequently expressed aim of enhancing transparency (Whyte and Macintosh, 2001). In the rest of the paper I will briefly outline two examples of ‘ethnographically -informed’ analysis of conversations in and around e-democracy systems, and implications for their evaluation. Conversation and technology in EDEN and AVANTI EDEN (Electronic Democracy European Network) and AVANTI (Added Value Access to New Technologies and services on the Internet) both involve public administrations and technology developers in project consortia , funded under the European Union’s Information Society Technologies programme. In both projects, which can only be briefly outlined in this short paper, the software products are intended to support and enhance communication between citizens and public administrations. The author is involved in both, in coordinating work on evaluation of the software. The development and evaluation of prototypes has been informed by participatory design (PD) methods, using scenarios and early prototypes to establish and test requirements with groups of ‘target users’. To an extent the projects’ aims overlap with PD, being concerned with ‘enhancing participation’ in decisionmaking (EDEN) and as citizens in an information society (AVANTI). EDEN is explicitly intended as an e-democracy application, meant to enhance the citizens’ voice in deliberations of elected representatives, and officers of the administration and planning professionals who act as its agents. AVANTI on the other hand is conceived as e-government, meant to enhance citizens’ capability to use the Internet for online transactions regarding services they would otherwise use in person, through interaction with an officer or intermediary (e.g. a librarian at a public enquiry desk). Both projects envisage more ‘natural’ interaction with systems, thus reducing barriers to citizens’ participation and addressing administrative concerns. The latter are largely framed in terms of process efficiency and service quality, with citizens as customers of e-government services. In EDEN enhanced access to, navigation, and comprehension of information on urban planning is meant to provide a basis for better-informed (more ‘deliberative’) contributions by the citizenry to public consultations on city planning. 1 EDEN involves city administrations in Antwerp, Bologna, Bremen, Vienna and Nisko (Poland), together with software providers Yana Research, Omega Generation, and Public Voice Lab; with support from the University of Bremen’s TZI, Bologna’s Osvaldo Piacentini Archive. AVANTI comprises city administrations in Edinburgh, Lewisham (UK), Ventspils (Latvia) and Kista (Sweden), with software providers Microsoft and Fujitsu Consulting. Both also include Napier University’s International Teledemocracy Centre as partners. CSCW@E-democracy Workshop, ECSCW ’03, Helsinki Sept. 15 2003 3 An important expectation in administrative (managerial) thinking is that ‘performance measures’ should be applied to e-democracy and e-government initiatives. So a focus of the evaluation approach has been to collaborate with the city administrations to develop evaluation criteria and indicators, to assess the prototypes against qualitative data and quantitative ‘satisfaction’ ratings on agreed usability and acceptability factors. Both projects have produced software toolkits to structure citizen-administration communications, giving roles to non-human actors that employ NLP (Natural Language Processing) capabilities:• In EDEN the non-human actors (software modules) are used inter-alia to take messages that citizen-users enter on a web-based mail interface, and ‘guess’ on the basis of their content which office mailbox they should go to in the city administration. • In AVANTI on the other hand, NLP is used to automatically produce responses delivered as text and voice by cartoon-like characters or ‘avatars’. A further point in common is that in both projects software systems have been conceived and designed from the perspective that computational linguistics, applied in NLP software, can meaningfully process human texts on the basis of the syntactic and semantic structures their analysis yields. In EDEN’s case the texts are enquiries that citizens would ordinarily make by email to public administration officers who handle enquiries, or enter into city website search facilities. AVANTI adds to this the notion that cartoon-like conversational agents can usefully mimic the part (again) played ordinarily by public administration officers who handle enquiries, by producing a response they might ordinarily give. How the projects frame those interactions and how they constitute ‘conversation’ (or not) is an interesting research question and, as shown in the rest of the paper, an evaluation concern. The limited ethnographic studies outlined here have informed both the evaluation criteria and the assessment of prototypes against them. Limited by time/resources and the still somewhat marginal role of ethnography in e-democracy development, ethnographic description has been ‘quick and dirty’. In EDEN it has been focused on translated interview notes and samples of email exchanges between citizen/users and city council officers. AVANTI has involved analysing video recordings of exchanges between citizen/users and ‘avatar’ characters, during ‘feedback sessions’. Participants are also asked to complete a questionnaire and offer their opinions on the software’s suitability for its purpose and use at public access points (in libraries for example). EDEN’s route from citizen to council office One of the EDEN tools is Address Guesser , one of 7 that when deployed in various combinations are meant to extend the resources and opportunities available to citizens, to make better-informed decisions on whether or not to give their views on city planning decisions to council officials. If that is what they decide, they may do so online in discussion fora that are structured around links to planning proposals and supporting information, or via direct contact to the office responsible for the issue they want to raise. The project assumes that having made that choice the resulting communication between citizen and public official can be done with less effort per citizen online. The (imminent) evaluation of these tools therefore needs to consider the difference they make to the effort required on the part of citizens and officials. As well as asking them, samples will be taken of online exchanges made using Address Guesser web-based mail interface. The evaluation criteria and indicators for this were partly informed by analysis of earlier samples of online email exchanges between citizens and local government officers. 2 See for example the Council of Europe’s current draft recommendations on the measurement of impact and cost-effectiveness of e-governance at http://www.coe.int/t/e/integrated_projects/democracy/02_Activities/01_e-governance/01_egovernance_draft_recs_v2.asp (August 2003) 3 This and other aspects of the EDEN project are discussed in more detail in Whyte and Macintosh (2003), and Westholm